Shader and material layers for rendering three-dimensional (3D) object data models

ABSTRACT

Methods and systems for material refinement for portions of a three-dimensional (3D) object data model are provided. An example method may include rendering a portion of a 3D object data model, and determining a first appearance metric between an appearance of the portion in the rendered view and a two-dimensional (2D) image. For one or more iterations, a modification to material properties associated with the portion may be determined based on the first appearance metric, and another view of the portion of the 3D object data model may be rendered. Also for the one or more iterations, another appearance metric between and an appearance of the portion in the rendered another view and the 2D image may be determined. Additionally, modified material properties for the portion that are associated with a minimum appearance metric of the one or more iterations may be stored for the 3D object data model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/558,910 filed on Nov. 11, 2011, the entirety of which is hereinincorporated by reference.

BACKGROUND

In computer graphics, three-dimensional (3D) modeling involvesgeneration of a representation of a 3D surface of an object. Therepresentation may be referred to as a 3D object data model, and can berendered or displayed as a two-dimensional image via 3D rendering ordisplayed as a three-dimensional image. 3D object data models representa 3D object using a collection of points in 3D space, connected byvarious geometric entities such as triangles, lines, curved surfaces,etc. Various techniques exist for generating 3D object data modelsutilizing point clouds and geometric shapes, for examples.

Being a collection of data, 3D models can be created by hand,algorithmically, or by scanning objects, for example. As an example, anartist may manually generate a 3D image of an object that can be used asthe 3D model. As another example, a given object may be scanned from anumber of different angles, and the scanned images can be combined togenerate the 3D image of the object. As still another example, an imageof an object may be used to generate a point cloud that can bealgorithmically processed to generate the 3D image.

3D object data models may include solid models that define a volume ofthe object, or may include shell or boundary models that represent asurface (e.g. the boundary) of the object. Because an appearance of anobject depends largely on an exterior of the object, boundaryrepresentations are common in computer graphics.

3D models are used in a wide variety of fields, and may be displayedusing a number of different types of interfaces. Example interfaces mayprovide functionality to enable interaction between a user and the 3Dmodels.

SUMMARY

In one example aspect, a method is provided that includes rendering aportion of a three-dimensional (3D) object data model of an object basedon material properties associated with the portion. The 3D object datamodel may include material properties associated with geometry of the 3Dobject data model. The method may also include using a computing deviceto determine a first appearance metric between an appearance of theportion in the rendered view and an appearance of the portion in atwo-dimensional (2D) image. According to the method, for one or moreiterations, a modification to one or more material properties associatedwith the portion may be determined based on the first appearance metric,and another view of the portion of the 3D object data model may berendered based on the modification. Also for the one or more iterations,another appearance metric between an appearance of the portion in the 2Dimage and an appearance of the portion in the rendered another view maybe determined. Additionally, the method may include storing the 3Dobject data model of the object having modified material properties forthe portion. The modified material properties may be associated with aminimum appearance metric of the one or more iterations.

In another example aspect, a computer-readable medium having storedtherein instructions, that when executed by a computing device, causethe computing device to perform functions is provided. The functions mayinclude rendering a portion of a three-dimensional (3D) object datamodel of an object based on material properties associated with theportion. The 3D object data model may include material propertiesassociated with geometry of the 3D object data model. The functions mayalso include determining a first appearance metric between an appearanceof the portion in the rendered view and an appearance of the portion ina two-dimensional (2D) image. According to the functions, for one ormore iterations, a modification to one or more material propertiesassociated with the portion may be determined based on the firstappearance metric, and another view of the portion of the 3D object datamodel may be rendered based on the modification. Also for the one ormore iterations, another appearance metric between an appearance of theportion in the 2D image and an appearance of the portion in the renderedanother view may be determined. Additionally, the functions may includestoring the 3D object data model of the object having modified materialproperties for the portion. The modified material properties may beassociated with a minimum appearance metric of the one or moreiterations.

In still another example aspect, a system is provided that includes arendering component, a refinement component, and a material component.The rendering component may be configured to render a view of a portionof a three-dimensional (3D) object data model of an object based onmaterial properties associated with geometry of the 3D object datamodel. The refinement component may be configured, for one or moreiterations, to determine an appearance metric between an appearance ofthe portion in a view that is rendered by the rendering component and anappearance of the portion in a two-dimensional (2D) image. Also for theone or more iterations, the refinement component may be configured todetermine a modification to one or more of the materials propertiesassociated with the portion based on the appearance metric, and providethe modification to the rendering component. The material component maybe configured to store the 3D object data model of the object havingmodified material properties for the portion. The modified materialproperties may be associated with a minimum appearance metric of the oneor more iterations.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates an example system for object data modeling.

FIG. 2 illustrates another example system for object data modeling.

FIG. 3 is a block diagram of an example method for material refinementfor a three-dimensional (3D) object data model.

FIG. 4 is a flow chart of an example method for material refinement fora three-dimensional (3D) object data model.

FIG. 5 is a conceptual illustration of an example rendered view of athree-dimensional (3D) object data model.

FIG. 6 is a functional block diagram illustrating an example computingdevice used in a computing system that is arranged in accordance with atleast some embodiments described herein.

FIG. 7 is a schematic illustrating a conceptual partial view of anexample computer program product that includes a computer program forexecuting a computer process on a computing device, arranged accordingto at least some embodiments presented herein.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying figures, which form a part hereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. The illustrative embodiments described in the detaileddescription, figures, and claims are not meant to be limiting. Otherembodiments may be utilized, and other changes may be made, withoutdeparting from the scope of the subject matter presented herein. It willbe readily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are explicitlycontemplated herein.

This disclosure may disclose, inter alia, methods and systems for shaderand material layers for rendering three-dimensional (3D) object datamodels. In some examples, a client device may receive a 3D object datamodel from a server and render a representation of the 3D object datamodel by executing a shader layer and a materials layer for variouscomponents of the 3D object data model. For instance, the 3D object datamodel may include components made of different types of materials, andthe components may be separated based on type of material. Additionally,individual types of materials may be assigned a given shader tofacilitate rendering the material. When a client device receives a 3Dobject data model, the client device may also receive shader andmaterial information for the various components of the 3D object datamodel, and render the various components of the 3D object data modelusing the respective shaders for each type of material.

In one example, a modification to material properties for a 3D objectdata model may be automatically determined. For instance, a 3D objectdata model may be provided with default or generic material properties,such as predetermined textures available within a 3D modeling programused to generate the 3D object data model. In some cases, the materialproperties provided with a 3D object data model may be modified suchthat when a representation of the 3D object data model is rendered, therepresentation matches an actual appearance of a product represented bythe 3D object data model.

According to an example method, a view of a portion of a 3D object datamodel may be rendered based on material properties associated with theportion. Subsequently, an appearance metric may be determined between anappearance of the portion in the rendered view and an appearance of theportion in a two-dimensional (2D) image. For example, the appearancemetric may be a numerical distance between a representation of a colorof the portion in the 2D image and a representation of a color of theportion in the rendered view. Based on the appearance metric, amodification to one or more of the material properties may bedetermined, and another view of the portion of the 3D object data modelmay be rendered based on the modified material properties. In oneinstance, modifying the material properties associated with the portionmay include selecting a different shader for the portion. The method mayfurther involve determining another appearance metric between therendered another view and the 2D image. According to the example method,material properties associated with a minimum appearance metric may bestored with the 3D object data model.

In some instances, additional modifications to the material propertiesmay be performed as part of an iterative method. For instance,information regarding whether a given modification increases ordecreases the appearance metric may be provided as feedback and used todetermine subsequent modifications to the material properties within afeedback loop. As another example, an expectation-maximization (EM)algorithm may be used to refine the material properties associated witha portion of the 3D object data model. Thus, in some examples, materialproperties associated with portions of a 3D object data model may berefined by comparing renderings of the 3D object data model with imagesof a product represented by the 3D object data model. In some instances,the methods for refinement of material properties described herein mayautomate the guess work and trial and metric associated with matching arendering of a 3D object data model to a 2D image of the object orproduct represented by the 3D object data model.

Referring now to the figures, FIG. 1 illustrates an example system 100for object data modeling. The system 100 includes an input source 102coupled to a server 104 and a database 106. The server 104 is also showncoupled to the database 106 and an output target 108. The system 100 mayinclude more or fewer components, and each of the input source 102, theserver 104, the database 106, and the output target 108 may comprisemultiple elements as well, or each of the input source 102, the server104, the database 106, and the output target 108 may be interconnectedas well. Thus, one or more of the described functions of the system 100may be divided up into additional functional or physical components, orcombined into fewer functional or physical components. In some furtherexamples, additional functional and/or physical components may be addedto the examples illustrated by FIG. 1.

Components of the system 100 may be coupled to or configured to becapable of communicating via a network (not shown), such as a local areanetwork (LAN), wide area network (WAN), wireless network (e.g., a Wi-Finetwork), or Internet, for example. In addition, any of the componentsof the system 100 may be coupled to each other using wired or wirelesscommunications. For example, communication links between the inputsource 102 and the server 104 may include wired connections, such as aserial or parallel bus, or wireless links, such as Bluetooth, IEEE802.11 (IEEE 802.11 may refer to IEEE 802.11-2007, IEEE 802.11n-2009, orany other IEEE 802.11 revision), or other wireless based communicationlinks.

The input source 102 may be any source from which a 3D object data modelmay be received. In some examples, 3D model acquisition (shape andappearance) may be achieved by working with venders or manufacturers toscan objects in 3D. For instance, structured light scanners may captureimages of an object and a shape of the object may be recovered usingmonochrome stereo cameras and a pattern projector. In other examples, ahigh-resolution DSLR camera may be used to capture images for colortexture information. In still other examples, a raw computer-aideddrafting (CAD) set of drawings may be received for each object. Thus,the input source 102 may provide a 3D object data model, in variousforms, to the server 104. As one example, multiple scans of an objectmay be processed into a merged mesh and assets data model, and providedto the server 104 in that form.

The server 104 includes a model builder 110, an object data modelprocessor 112, a semantics and search index 114, a graphics library 116,a shader application 118, a materials application 120, and an objectdata model renderer/viewer 122. Any of the components of the server 104may be coupled to each other. In addition, any components of the server104 may alternatively be a separate component coupled to the server 104.The server 104 may further include a processor and memory includinginstructions executable by the processor to perform functions of thecomponents of the server 104, for example.

The model builder 110 receives the mesh data set for each object fromthe input source 102, which may include a data set defining a densesurface mesh geometry, and may generate an animated model of the objectin 3D. For example, the model builder 110 may perform coherent textureunwrapping from the mesh surface, and determine textures of surfacesemulated from the geometry.

The object data model processor 112 may also receive the mesh data setfor each object from the input source 102 and generate display meshes.For instance, the scanned mesh images may be decimated (e.g., from 5million to 120,000 surfaces) utilizing texture-preserving decimation.Texture map generation can also be performed to determine color texturefor map rendering. Texture map generation may include using the meshdata sets (H) that have colors but no ultraviolet (UV) unwrapping togenerate a mesh (D) with UV unwrapping but no colors. As an example, fora single output texture pixel of an image, processing may include, for agiven point in UV, determine a triangle in the mesh's UV mapping (D),and using triangle-local coordinates, move to an associated 3D point onthe mesh. A bidirectional ray may be cast along the triangle's normal tointersect with the mesh (H), and color, normal and displacement may beused for an output. To generate an entire texture image, each pixel inthe image can be processed.

The semantics and search index 114 may receive captured images orprocessed images that have been decimated and compressed, and mayperform texture resampling and also shape-based indexing. For example,for each object, the semantics and search index 114 may index or labelcomponents of the images (e.g., per pixel) as having a certain texture,color, shape, geometry, attribute, etc. The semantics and search index114 may receive the 3D object data model file or files comprising the 3Dobject data model from the model builder 110 or the object data modelprocessor 112, and may be configured to label portions of the file oreach file individually with identifiers related to attributes of thefile.

In some examples, the semantics and search index 114 may be configuredto provide annotations for aspects of the 3D object data models. Forinstance, an annotation may be provided to label or index aspects ofcolor, texture, shape, appearance, description, function, etc., of anaspect of a 3D object data model. Annotations may be used to label anyaspect of an image or 3D object data model, or to provide any type ofinformation. Annotations may be performed manually or automatically. Inexamples herein, an annotated template of an object in a givenclassification or category may be generated that includes annotations,and the template may be applied to all objects in the givenclassification or category to apply the annotations to all objects.

The graphics library 116 may include a WebGL or OpenGL mesh compressionto reduce a mesh file size, for example. The graphics library 116 mayprovide the 3D object data model in a form for display on a browser, forexample. In some examples, a 3D object data model viewer may be used todisplay images of the 3D objects data models. The 3D object data modelviewer may be implemented using WebGL within a web browser, or OpenGL,for example.

The shader application 118 may be configured to apply a shader toportions of the 3D object data model file or files of the 3D object datamodel according to the indexes of the file (as labeled by the semanticsand search index 114) to generate a 3D image. The shader application 118may be executed to apply a shader from a number of shaders according tothe indexes of the file. The shader may include information related totexture, color, appearance, etc., of a portion of the 3D image.

In one example, the shader application 118 may be executed to render animage with shading attributes as defined by indexes of the files. Forexample, objects with multiple surfaces may have different attributesfor each surface, and the shader application 118 may be executed torender each surface accordingly.

The materials application 120 may be configured to apply a material toportions of the 3D object data model file or to files of the 3D objectdata model according to the indexes of the file (as labeled by thesemantics and search index 114) to generate a 3D image. The materialsapplication 120 may be executed to apply a material from a number ofmaterials according to the indexes of the file. The materialsapplication may apply any material, such as leather, metal, wood, etc.,so as to render an appearance of a portion of the 3D image.

In one example, the materials application 120 may access a database thatincludes information regarding a number of reference materials (e.g.,brass, fur, leather), and objects with multiple materials may beseparated into distinct portions. As an example, a hood on a car mayinclude a hood ornament, and the hood may be painted while the ornamentmay have a chrome finish. The materials application 120 and the shaderapplication 118 can be executed to identify two separate materials(e.g., the painted hood and the chrome hood ornament) and render eachmaterial with an appropriate shader.

The object data model renderer/viewer 122 may receive the 3D object datamodel file or files and execute the shader application 118 and thematerials application 120 to render a 3D image, representation, or viewof the 3D object data model or a portion of the 3D object data model.

The database 106 may store all data sets for a 3D object data model inany number of various forms from raw, captured data to processed datafor display.

The output target 108 may include a number of different targets, such asa webpage on the Internet, a search engine, a database, etc. The outputtarget 108 may include a 3D object data model viewer that enablesproduct advertisements or product searches based on the 3D object datamodel.

In examples herein, the system 100 may be used to acquire data of anobject, process the data to generate a 3D object data model, and renderthe 3D object data model for display.

FIG. 2 illustrates another example system 200 for object data modeling.The system 200 includes the input source 102 coupled to the server 104,which is coupled to a client device 124. The input source 102 and theserver 104 may be as described in FIG. 1. The client device 124 mayreceive outputs from any of the components of the server 124, and may beconfigured to render a 3D image.

The client device 124 includes an object data model renderer/viewer 126,a shader application 128, and a materials application 130. The objectdata model renderer/viewer 126, the shader application 128, and thematerials application 130 may all be configured as described withrespect to the object data model renderer/viewer 122, the materialsapplication 120, and the shader application 118 of the server 104 asdescribed with respect to FIG. 1.

In some examples, the client device 124 may receive the 3D object datamodel file or files from the server 104 and render a view of the objectby executing the shader application 128 and the materials application130. When executing the shader application 128 and the materialsapplication 130, the client device 124 may access separate databases toretrieve appropriate shader and material information to apply to theimage, access the server 104 to receive appropriate shader and materialinformation from the shader application 118 and the materialsapplication 120, or may store information locally regarding theappropriate shader and material information to apply.

In some examples, the client device 124 may receive the 3D object datamodel file or files from the server 104 and render a 3D image of theobject. In other examples, the server 104 may render a 3D image of theobject and stream the 3D image to the client device 124 for display.

In some examples, components of the client device 124 may be configuredto refine material properties associated with a 3D object data modelbased on a comparison between a rendered view of a portion of the 3Dobject data model and a two-dimensional image of a product that isrepresented by the 3D object data model. For instance, the object datamodel render/viewer 126 may, in some examples, include a renderingcomponent that is configured to render a view of a portion of a 3Dobject data model of an object. The view of the portion of the 3D objectdata model may be rendered by the rendering component based on materialproperties that are associated with geometry (e.g., coordinates ofvertices of a polygonal mesh) of the 3D object data model. Additionally,a refinement component of the client device 124 may be configure todetermine an appearance metric between an appearance of the portion in arendered view and an appearance of the portion in a 2D image. Based onthe appearance metric, the refinement component may also be configuredto determine a modification to the material properties associated withthe object, and provide the modified material properties to therendering component. Also, a material component of the client device 124may be configured to store the 3D object data model of the object havingmodified material properties for the portion. The modified materialproperties may be material properties which yield the minimum appearancemetric of one or more determined appearance metrics, for example.

Although the example is described as performed by the client device 124,in some instances, the material refinement may be performed bycomponents of the server 104, or a combination of components of theserver 104 and the client device 124. Additional details of therefinement of the material properties are further described below withrespect to FIGS. 3 and 4.

FIG. 3 is a block diagram of an example method 300 for materialrefinement for a three-dimensional (3D) object data model. Method 300shown in FIG. 3 presents an embodiment of a method that, for example,could be used with the system 100 or the system 200, for example, andmay be performed by a device, such as any of the components illustratedin FIG. 1 or 2. Method 300 may include one or more operations,functions, or actions as illustrated by one or more of blocks 302-312.Although the blocks are illustrated in a sequential order, these blocksmay also be performed in parallel, and/or in a different order thanthose described herein. Also, the various blocks may be combined intofewer blocks, divided into additional blocks, and/or removed based uponthe desired implementation.

In addition, for the method 300 and other processes and methodsdisclosed herein, the block diagram shows functionality and operation ofone possible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor orcomputing device for implementing specific logical functions or steps inthe process. The program code may be stored on any type of computerreadable medium, for example, such as a storage device including a diskor hard drive. The computer readable medium may include non-transitorycomputer readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and Random Access Memory (RAM). The computer readablemedium may also include non-transitory media, such as secondary orpersistent long term storage, like read only memory (ROM), optical ormagnetic disks, compact-disc read only memory (CD-ROM), for example. Thecomputer readable media may also be any other volatile or non-volatilestorage systems. The computer readable medium may be considered acomputer readable storage medium, for example, or a tangible storagedevice.

In addition, for the method 300 and other processes and methodsdisclosed herein, each block in FIG. 3 may represent circuitry that iswired to perform the specific logical functions in the process.

Initially, at block 302, the method 300 includes rendering a view of aportion of a 3D object data model based on material propertiesassociated with the portion. In one example, a 3D object data model thatis a representation of a product, such as a mobile phone or other typeof electronic device, a shoe, article of clothing, toy, or any type ofobject, may be received. The 3D object data model may include multiplecomponents, each of which may be rendered using separate material assetsand shader assets. For example, a mobile phone may include a displayscreen and a frame. The display screen may be rendered based on materialattributes and shaders associated with glass, for instance, while theframe may be rendered using a shader that is applied to types ofplastics or metals. As another example, the 3D object data model may bea boot, and the boot may include a rubber sole, a leather upper, and ametal clasp (each of which may be rendered based on separate materialproperties).

The 3D object data model may include material properties that areassociated with geometry of the 3D object data model. In one example,the 3D object data model may include a list of geometry coordinates ofvertices of a polygonal mesh having pointers to material attributesassociated with the geometry coordinates. The material properties mayidentify appearance attributes or material shaders for the geometrycoordinates. For example, the appearance attributes may be lightingvalues, texture coordinates, and/or colors that are provided as inputsfor a material shader. The material shader may be configured todetermine a pixel color or other rendering effect based on theappearance attributes. In another instance, the material properties maybe texture maps, such as diffuse maps, bump maps, opacity maps, glowmaps, or specular maps.

In one example, the portion of the 3D object data model may be acomponent of the 3D object data model having common material properties,such as a common texture map or material shader. For instance, if the 3Dobject data model is a boot, the portion may be a rubber sole of theboot (or a portion of the rubber sole). A view of solely the portion maybe rendered, or a view that includes the portion as well as otherportions of the 3D object data model may be rendered. In one instance, aview of 3D object data model in which the portion is visible may berendered to match an orientation of a product that is represented by the3D object data model in a 2D image of the product.

At block 304, the method 300 includes determining a first appearancemetric between an appearance of the portion in the rendered view and anappearance of the portion in a 2D image. For instance, a manufacturer ofa product may provide a 2D image of a product, or multiple 2D images foreach portion or type of material/texture of the product. In one example,a digital 2D image of the product, or the portion of the product in therendered view, may be compared with the rendered view. Any number ofcomputer vision techniques may be employed by a computing device inorder to determine an appearance metric between an appearance of theportion in the rendered view and appearance of the portion in the 2Dimage.

As an example, a numerical distance may be determined between arepresentation of a color of the portion in the 2D image and arepresentation of a color of the portion in the rendered view. Thenumerical distance may be a difference between an average pixel color ofthe portion in the 2D image and an average pixel color of the portion inthe rendered view. For instance, an average pixel color for a portion ofthe 2D image or rendered view may be determined by averaging the valuesof a center pixel with values of pixels surrounding the pixel. Thecomputing device may be able to determine red, green, and blue (RGB)components, hue, saturation, lightness (HSL) components, and/or hue,saturation, value (HSV) components of a pixel or group of pixels of therendered view and 2D image by digitally analyzing the rendered view andthe 2D image. Subsequently the color components may be converted to anabsolute color space such that the colors may be numerically compared.The converted colors may be subtracted or otherwise compared todetermine a level of match between the appearance of the portion in the2D image and the rendered view. For instance, two colors that are closetogether within the color space may be more similar than two colors thatare separated by a larger numerical distance in the color space. Othermethods and techniques for comparing colors of the appearance of theportion in the rendered view and the 2D image are also possible, and theexample is not meant to be limiting.

At block 306, the method 300 includes based on the first appearancemetric, determining a modification to one or more of the materialproperties. In one example, the material properties may include a givenshader that is configured to determine the pixel color of the portion inthe rendered view. A different shader for the portion may be selectedfrom a database, for example. The database may include multiple shadersfor different types of materials, such as leather, wood, metal, plastic,rubber, etc. In some instances, a given type of material may includemultiple shaders, and a different shader may be selected from within theshaders for the type of material. For example, a different shader mayprovide a different type of shading or may include bump mapping ortranslucency effects.

In further examples, when two or more shaders exist for a given type ofmaterial for a portion to be rendered, an assignment history can beaccessed to make a selection. For example, based on past usage of shaderof same, similar or related objects or regions of objects, the sameshader may be used and selected for rendering the portion.

In another example, appearance attributes associated with the portion ofthe 3D object data model may be modified. For example, the portion ofthe 3D object data model may include a base color, texture coordinates,and lighting information, among other parameters, which are used by ashader to determine a final pixel color that is rendered or displayed ona screen. Any of the appearance attributes may be modified based on theappearance metric.

At block 308, the method 300 includes rendering another view of theportion of the 3D object data model based on the modification to thematerial properties. For instance after modified appearance attributesor a new shader has been selected, another view of the portion of the 3Dobject data model may be rendered. In some instances, the portion isrendered having the same orientation and viewpoint as the first renderedview.

At block 310, the method 300 includes determining another appearancemetric between an appearance of the portion in the 2D image and anappearance of the portion in the rendered another view. The appearancemetric may be determined by a computing device using the same techniqueas performed at block 304. For example, the appearance metric may be anumerical distance between the representation of the color of theportion in the 2D image and the representation of the color of theportion in the another rendered view.

In some instances, as part of the method 300, blocks 306, 308, and 310are performed for one or more iterations such that multiple additionalappearance metrics associated with different modified materialproperties are determined. At block 312, the method 300 includes storingthe 3D object data model of the object having modified materialproperties for the portion that are associated with a minimum appearancemetric. For example, modified material properties associated with aminimum appearance metric over the one or more iterations may beidentified and stored with the 3D objet data model. Additionally, themethod 300 may be repeated for additional portions of the 3D object datamodel such that material properties associated with other portions ofthe 3D object data model that have separate material types may berefined and modified. Thus, the material properties associated with the3D object data model may be refined such that the appearances ofportions of the 3D object data model more closely match an appearance ofthe product that is represented by the 3D object data model.

FIG. 4 is a flow chart 400 of an example method for material refinementfor a three-dimensional (3D) object data model. The example method shownin FIG. 4 presents an embodiment of a method that could be used by thesystem 100 of FIG. 1 or the system 200 of FIG. 2, or components of thesystem 100 or system 200, for example.

The example method may include one or more operations, functions, oractions as illustrated by blocks of the flow chart. Although the blocksare illustrated in a sequential order, these blocks may also beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed from the flow chart,based upon the desired implementation of the method. Each block mayrepresent a module, a segment, or a portion of program code, whichincludes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Inaddition, each block in FIG. 4 may represent circuitry that is wired toperform the specific logical functions in the process.

Initially, at block 402, a 2D image of a product and a 3D object datamodel that is a representation of the product may be received. Forexample, the 2D image may be an image of a portion of the product or thewhole product captured by a camera. The 3D object data model may havebeen acquired by scanning the product to determine material and geometryinformation, or the 3D object data model may have been provided by amanufacturer or engineer who has labeled portions of the 3D object datamodel as being of a particular type of material (e.g., wood, leather,glass, plastic, etc.). At block 404, a view of a portion of the 3Dobject data model may be rendered. The portion may be a portion that islabeled as being of a particular type of material, such as a type ofmaterial pictured in the 2D image.

At block 406, an appearance metric between an appearance of the portionin the rendered view and an appearance of the portion in the 2D imagemay be determined. As described previously, the appearance metric may bea numerical distance in a color space between a representation of thecolor of the portion in the 2D image and a representation of the colorof the portion in the rendered view.

Moreover, at block 408, a decision may be made based on a relationshipbetween the appearance metric and an appearance threshold. In oneexample, if the appearance metric is less than an appearance threshold,the method may proceed to block 410, while if the appearance metric isgreater than the appearance threshold, the method may proceed to block412.

At block 412, modified material properties may be determined for theportion of the 3D object data model. For instance, a different shaderfor the portion may be selected or appearance attributes that are inputsto a material shader may be modified based on the appearance metric.Following, another view of the portion may be rendered at block 404, andanother appearance metric may be determined between the rendered anotherview and the 2D image at block 406. Additionally, at block 408, theanother appearance metric may be compared to the threshold to determinewhether the another appearance metric is less than the threshold. If theanother appearance metric is greater than the threshold, the method mayproceed to block 412, where another iteration of modifying materialproperties is performed. If the another appearance metric is less thanthe threshold, the method may proceed to block 410, where the modifiedmaterial properties that yielded the another appearance metric may bestored as material properties for the portion of the 3D object datamodel.

While the flow chart 400 conceptually illustrates one example iterativemethod of refining material properties for a portion(s) of a 3D objectdata model, another approach to refining the material properties of the3D object data model may be similar to an expectation-maximization (EM)algorithm. For instance, an EM algorithm is an efficient iterativeprocedure to compute the maximum likelihood estimate in the presence ofmissing or hidden data.

According to the EM algorithm, each iteration includes an E-step and anM-step. In the E-step, an expectation of a log-likelihood evaluatedusing a current estimate of missing model parameters is found. Alikelihood may indicate how likely a parameter value is in light of anobserved outcome. In the following M-step, parameters maximizing theexpected log-likelihood found in the previous E-step are found. Theparameters are then used for the estimate of the log-likelihood in thenext E-step. It can be shown that repeating the iterations leads toconvergence of the missing parameter(s) to fixed values. In the contextof the material properties of a 3D object data model and a 2D image, theEM algorithm may be used to estimate appearance attributes which areinputs to a material shader and yield a pixel color represented in the2D image. For instance, the missing parameters may be a base pixelcolor, texture coordinates, and/or lighting information, while theobserved data may be a pixel color in the 2D image. Thus, the EMalgorithm may be used to infer what appearance attributes may be used togenerate a pixel color matching the 2D image pixel color with theshader. Alternatively, the EM algorithm may be used to adjust internalparameters of a shader that are used to determine a pixel color based onfixed input appearance attributes.

FIG. 5 is a conceptual illustration 500 of an example rendered view of athree-dimensional (3D) object data model. As shown in FIG. 5, the 3Dobject data model may represent a chair. In some instances, a repeatingpattern within the 3D object data model may be identified. For example,the 3D object data model may be segmented into different portions basedon shaders used to render adjacent points of the 3D object data model,or portions of the 3D object data model may be segmented and labeled bya manufacturer as separate portions. Within a portion, a pattern ofrecurring elements may be identified, and a recurring element of thepattern may be selected as an area of the 3D object data model to bematched against a 2D image. For instance, the pattern may consist ofparallel lines of alternating color, adjacent polygons of differentcolors, interwoven strands of different colors, etc.

By selecting an area of the portion to match against the 2D image,rather than the entire portion, the refinement of the materialproperties associated with the 3D object data model may be improved. Forexample, after material properties associated with the portion have beenmodified, according to the method 300, for example, the materialproperties associated with the selected area may be applied to otherrecurring elements of the identified pattern throughout the portion.

As conceptually illustrated in FIG. 5, a first area 502 may be a portionof a cushion of the chair having a cotton pattern of woven thread. Afterthe material properties for the first area 502 have been modified, thematerial properties may be similarly modified for the remainder of thecushion of the chair. For instance, a new shader may be selected for thefirst area 502, and the new shader may also be applied to the remainderof the cushion. Other examples of a selected area include a second area504 that is part of a lower frame of the chair and a third area 506 thatis part of an upper frame of the chair. Thus, the portion of the 3Dobject data model that is matched against the 2D image may be a selectedarea of a component of the 3D object data model.

FIG. 6 is a functional block diagram illustrating an example computingdevice 600 used in a computing system that is arranged in accordancewith at least some embodiments described herein. The computing device600 may be a personal computer, mobile device, cellular phone,touch-sensitive wristwatch, tablet computer, video game system, orglobal positioning system, and may be implemented to provide a methodfor material refinement for 3D object data models as described in FIGS.1-5. In a basic configuration 602, computing device 600 may typicallyinclude one or more processors 610 and system memory 620. A memory bus630 can be used for communicating between the processor 610 and thesystem memory 620. Depending on the desired configuration, processor 610can be of any type including but not limited to a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination thereof. A memory controller 615 can also be used with theprocessor 610, or in some implementations, the memory controller 615 canbe an internal part of the processor 610.

Depending on the desired configuration, the system memory 620 can be ofany type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. System memory 620 may include one or more applications 622, andprogram data 624. Application 622 may include a material algorithm 623that is arranged to provide inputs to the electronic circuits, inaccordance with the present disclosure. Program data 624 may includecontent information 625 that could be directed to any number of types ofdata. In some example embodiments, application 622 can be arranged tooperate with program data 624 on an operating system.

Computing device 600 can have additional features or functionality, andadditional interfaces to facilitate communications between the basicconfiguration 602 and any devices and interfaces. For example, datastorage devices 640 can be provided including removable storage devices642, non-removable storage devices 644, or a combination thereof.Examples of removable storage and non-removable storage devices includemagnetic disk devices such as flexible disk drives and hard-disk drives(HDD), optical disk drives such as compact disk (CD) drives or digitalversatile disk (DVD) drives, solid state drives (SSD), and tape drivesto name a few. Computer storage media can include volatile andnonvolatile, non-transitory, removable and non-removable mediaimplemented in any method or technology for storage of information, suchas computer readable instructions, data structures, program modules, orother data.

System memory 620 and storage devices 640 are examples of computerstorage media. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing device 600.Any such computer storage media can be part of computing device 600.

Computing device 600 can also include output interfaces 650 that mayinclude a graphics processing unit 652, which can be configured tocommunicate to various external devices such as display devices 660 orspeakers via one or more A/V ports or a communication interface 670. Thecommunication interface 670 may include a network controller 672, whichcan be arranged to facilitate communications with one or more othercomputing devices 680 over a network communication via one or morecommunication ports 674. The communication connection is one example ofa communication media. Communication media may be embodied by computerreadable instructions, data structures, program modules, or other datain a modulated data signal, such as a carrier wave or other transportmechanism, and includes any information delivery media. A modulated datasignal can be a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communication media can includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, radio frequency (RF), infrared (IR) andother wireless media.

Computing device 600 can be implemented as a portion of a small-formfactor portable (or mobile) electronic device such as a cell phone, apersonal data assistant (PDA), a personal media player device, awireless web-watch device, a personal headset device, an applicationspecific device, or a hybrid device that include any of the abovefunctions. Computing device 600 can also be implemented as a personalcomputer including both laptop computer and non-laptop computerconfigurations.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a non-transitorycomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. FIG. 7 is aschematic illustrating a conceptual partial view of an example computerprogram product 700 that includes a computer program for executing acomputer process on a computing device, arranged according to at leastsome embodiments presented herein.

In one embodiment, the example computer program product 700 is providedusing a signal bearing medium 701. The signal bearing medium 701 mayinclude one or more programming instructions 702 that, when executed byone or more processors may provide functionality or portions of thefunctionality described above with respect to FIGS. 1-6. In someexamples, the signal bearing medium 701 may encompass acomputer-readable medium 703, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 701 mayencompass a computer recordable medium 704, such as, but not limited to,memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations,the signal bearing medium 901 may encompass a communications medium 705,such as, but not limited to, a digital and/or an analog communicationmedium (e.g., a fiber optic cable, a waveguide, a wired communicationslink, a wireless communication link, etc.). Thus, for example, thesignal bearing medium 701 may be conveyed by a wireless form of thecommunications medium 705 (e.g., a wireless communications mediumconforming with the IEEE 802.11 standard or other transmissionprotocol).

The one or more programming instructions 702 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computing device 600 of FIG. 6may be configured to provide various operations, functions, or actionsin response to the programming instructions 702 conveyed to thecomputing device 600 by one or more of the computer readable medium 703,the computer recordable medium 704, and/or the communications medium705.

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A method, comprising: rendering a view of aportion of a three-dimensional (3D) object data model of an object basedon material properties associated with the portion, wherein the 3Dobject data model comprises material properties associated with geometryof the 3D object data model; determining, using a computing device, afirst appearance metric between an appearance of the portion in therendered view and an appearance of the portion in a two-dimensional (2D)image by comparing the appearance of the portion in the rendered viewwith the appearance of the portion in the 2D image; for one or moreiterations: based on the first appearance metric, determining amodification to one or more of the material properties associated withthe portion; rendering another view of the portion of the 3D object datamodel based on the modification to the material properties associatedwith the portion; and determining, using the computing device, anotherappearance metric between an appearance of the portion in the 2D imageand an appearance of the portion in the rendered another view; andstoring the 3D object data model of the object having modified materialproperties for the portion, wherein the modified material properties areassociated with a minimum appearance metric of the one or moreiterations.
 2. The method of claim 1, wherein the first appearancemetric comprise a numerical distance between a representation of a colorof the portion in the 2D image and a representation of a color of theportion in the rendered view.
 3. The method of claim 1, wherein thematerial properties associated with the portion comprise appearanceattributes, and the method further comprises rendering the another viewof the portion using a given shader, wherein the shader is configured todetermine rendering effects for the portion based on the appearanceattributes.
 4. The method of claim 3, wherein determining a modificationto the material properties associated with the portion comprisesdetermining another shader for the portion.
 5. The method of claim 4,further comprising selecting the another shader from a plurality ofshaders of a material database.
 6. The method of claim 3, whereindetermining a modification to the material properties associated withthe portion comprises determining modified appearance attributes for theportion.
 7. The method of claim 1, further comprising: identifying,using the computing device, a repeating pattern within the materialproperties associated with the 3D object data model; determining theportion of the 3D object data model based on the repeating pattern,wherein the portion comprises a recurring element of the repeatingpattern; and applying the modified material properties to one or moreadditional portions of the 3D object data model for the stored 3D objectdata model, wherein the additional portions correspond to additionalelements of the repeating pattern.
 8. A non-transitory computer-readablemedium having stored therein instructions, that when executed by acomputing device, cause the computing device to perform functionscomprising: rendering a view of a portion of a three-dimensional (3D)object data model of an object based on material properties associatedwith the portion, wherein the 3D object data model comprises materialproperties associated with geometry of the 3D object data model;determining a first appearance metric between an appearance of theportion in the rendered view and an appearance of the portion in atwo-dimensional (2D) image by comparing the appearance of the portion inthe rendered view with the appearance of the portion in the 2D image;for one or more iterations: based on the first appearance metric,determining a modification to one or more of the material propertiesassociated with the portion; rendering another view of the portion ofthe 3D object data model based on the modification to the materialproperties associated with the portion; and determining anotherappearance metric between an appearance of the portion in the 2D imageand an appearance of the portion in the rendered another view; andstoring the 3D object data model of the object having modified materialproperties for the portion, wherein the modified material properties areassociated with a minimum appearance metric of the one or moreiterations.
 9. The non-transitory computer-readable medium of claim 8,wherein the first appearance metric comprise a numerical distancebetween a representation of a color of the portion in the 2D image and arepresentation of a color of the portion in the rendered view.
 10. Thenon-transitory computer-readable medium of claim 8, wherein the materialproperties associated with the portion comprise appearance attributes,and wherein the functions further comprise rendering the another view ofthe portion using a given shader, wherein the shader is configured todetermine rendering effects for the portion based on the appearanceattributes.
 11. The non-transitory computer-readable medium of claim 10,wherein determining a modification to the material properties associatedwith the portion comprises determining another shader for the portion.12. The non-transitory computer-readable medium of claim 11, wherein thefunctions further comprise selecting the another shader from a pluralityof shaders of a material database.
 13. The non-transitorycomputer-readable medium of claim 10, wherein determining a modificationto the material properties associated with the portion comprisesdetermining modified appearance attributes for the portion.
 14. Thenon-transitory computer-readable medium of claim 8, wherein thefunctions further comprise: identifying a repeating pattern within thematerial properties associated with the 3D object data model;determining the portion of the 3D object data model based on therepeating pattern, wherein the portion comprises a recurring element ofthe repeating pattern; and applying the modified material properties toone or more additional portions of the 3D object data model for thestored 3D object data model, wherein the additional portions correspondto additional elements of the repeating pattern.
 15. A system,comprising: a rendering component, the rendering component configured torender a view of a portion of a three-dimensional (3D) object data modelof an object based on material properties associated with geometry ofthe 3D object data model; a refinement component, the refinementcomponent configured to perform the following functions for one or moreiterations: determine an appearance metric between an appearance of theportion in a view that is rendered by the rendering component and anappearance of the portion in a two-dimensional (2D) image by comparingthe appearance of the portion in the view that is rendered with theappearance of the portion in the 2D image; determine a modification toone or more of the material properties associated with portion based onthe appearance metric; and provide the modification to the renderingcomponent; and a material component, the material component configuredto store the 3D object data model of the object having modified materialproperties for the portion, wherein the modified material properties areassociated with a minimum appearance metric of the one or moreiterations.
 16. The system of claim 15, wherein the appearance metriccomprise a numerical distance between a representation of a color of theportion in the 2D image and a representation of a color of the portionin the rendered view.
 17. The system of claim 15, wherein the materialproperties associated with the portion comprise appearance attributes,and wherein the rendering component is further configured to render theview of the portion using a given shader, wherein the shader isconfigured to determine rendering effects for the portion based on theappearance attributes.
 18. The system of claim 17, wherein therefinement component determining a modification to the materialproperties associated with the portion comprises determining anothershader for the portion.
 19. The system of claim 18, wherein therefinement component is further configured to select the another shaderfrom a plurality of shaders of a material database.
 20. The system ofclaim 15, wherein the material component is further configured to:identify a repeating pattern within the material properties associatedwith the 3D object data model, wherein the portion of the 3D object datamodel comprises a recurring element of the repeating pattern; and applythe modified material properties to one or more additional portions ofthe 3D object data model for the stored 3D object data model, whereinthe additional portions correspond to additional elements of therepeating pattern.